Add subscription/defer observability attributes and metrics by rohan-b99 · Pull Request #8858 · apollographql/router

rohan-b99 · 2026-02-03T17:46:30Z

Adds new span attributes and metrics to improve observability of streaming responses.

Span attributes:

apollo.subscription.end_reason: Records the reason a subscription was terminated. Possible values are server_close, subgraph_error, heartbeat_delivery_failed, client_disconnect, schema_reload, and config_reload.
apollo.defer.end_reason: Records the reason a deferred query ended. Possible values are completed (all deferred chunks were delivered successfully) and client_disconnect (the client disconnected before all deferred data was delivered).

Both attributes are added dynamically to router spans only when relevant (i.e., only on requests that actually use subscriptions or @defer), rather than being present on every router span.

Metrics:

The following counters are emitted when a subscription terminates:

apollo.router.operations.subscriptions.stream_end (attributes: subgraph.service.name): The subgraph gracefully closed the stream.
apollo.router.operations.subscriptions.subgraph_error (attributes: subgraph.service.name): The subscription terminated unexpectedly due to a subgraph error (e.g. process killed, network drop).
apollo.router.operations.subscriptions.client_disconnect (attributes: apollo.client.name): The client disconnected before the subscription ended.
apollo.router.operations.subscriptions.heartbeat_delivery_failed (attributes: apollo.client.name): A heartbeat could not be delivered to the client.
apollo.router.operations.subscriptions.schema_reload: The subscription was terminated because the router schema was updated.
apollo.router.operations.subscriptions.config_reload: The subscription was terminated because the router configuration was updated.

The following counter is emitted when a subscription request is rejected:

apollo.router.operations.subscriptions.rejected.limit: A new subscription request was rejected because the router has reached its max_opened_subscriptions limit.

Checklist

Complete the checklist (and note appropriate exceptions) before the PR is marked ready-for-review.

Exceptions

Note any exceptions here

Notes

It may be appropriate to bring upcoming changes to the attention of other (impacted) groups. Please endeavour to do this before seeking PR approval. The mechanism for doing this will vary considerably, so use your judgement as to how and when to do this. ↩
Configuration is an important part of many changes. Where applicable please try to document configuration examples. ↩
A lot of (if not most) features benefit from built-in observability and debug-level logs. Please read this guidance on metrics best-practices. ↩
Tick whichever testing boxes are applicable. If you are adding Manual Tests, please document the manual testing (extensively) in the Exceptions. ↩

apollo-librarian · 2026-02-03T17:46:38Z

✅ Docs preview ready

The preview is ready to be viewed. View the preview

File Changes

0 new, 2 changed, 0 removed

* graphos/routing/(latest)/observability/router-telemetry-otel/enabling-telemetry/standard-instruments.mdx
* graphos/routing/(latest)/operations/subscriptions/configuration.mdx

Build ID: adf329da55809814b64cc9df
Build Logs: View logs

URL: https://www.apollographql.com/docs/deploy-preview/adf329da55809814b64cc9df

…t-payload-errors

…errors' of https://github.com/apollographql/router into rohan-b99/subscription-observability-heartbeat-payload-errors

carodewig

This is looking great! One comment/question below

apollo-router/src/protocols/multipart.rs

carodewig

I've pushed a commit which does this (although the reason is now stored in a enum in the Multipart struct) - let me know if that's better or if you had something else in mind

I think it's easier to understand now, but happy to be overruled!

carodewig · 2026-02-11T16:55:52Z

apollo-router/src/protocols/multipart.rs

                        response.has_next.unwrap_or(false) || response.subscribed.unwrap_or(false);
+
+                    // Check for reload-related termination before errors are moved
+                    let end_reason = Self::detect_reload_end_reason(&response.errors);


Maybe I'm missing something - but is this variable ever used / stored?

It was used further down but only for subscriptions if the connection was no longer open - I did a small refactor to make it optional as at this point we don't know for sure if it has closed, the response gets moved a bit lower down though so I think it does need to be checked here

…t-payload-errors

rohan-b99 · 2026-02-12T16:21:24Z

Hi @theJC, just tagging you here to see if you had any feedback on this implementation - I understand the initial request for more observability on heartbeats/disconnections for subscriptions/@defer requests came from you a while ago, looking to see if these two new attributes align with expectations

pragl

I've left some suggestions on the docs changes to align with our style guide. Please consider them when you get the chance. Thanks!

docs/source/routing/operations/subscriptions/configuration.mdx

Co-authored-by: Parker <parker.ragland@apollographql.com>

theJC · 2026-02-14T06:57:42Z

apollo-router/src/protocols/multipart.rs

+    /// Used to detect if a heartbeat was the last thing sent before connection closed.
+    heartbeat_pending: bool,
+    /// The span captured at creation time, used to record attributes on connection close.
+    span: Span,


recommend naming this creation_span rather than span

theJC · 2026-02-14T07:09:09Z

apollo-router/src/protocols/multipart.rs

+        let end_reason = self.end_reason.take().unwrap_or_else(|| match self.mode {
+            ProtocolMode::Subscription => {
+                // Stream wasn't terminated properly - determine the reason
+                let reason = if self.heartbeat_pending {
+                    // Heartbeat was the last thing sent - likely failed to deliver
+                    SubscriptionEndReason::HeartbeatDeliveryFailed
+                } else {
+                    // Connection closed after a message was sent
+                    SubscriptionEndReason::ClientDisconnect
+                };
+                EndReason::Subscription(reason)
+            }
+            ProtocolMode::Defer => {
+                // Defer stream wasn't terminated properly - client disconnected
+                EndReason::Defer(DeferEndReason::ClientDisconnect)
+            }


Would it be cleaner if this complex logic be extracted into a function infer_abnormal_termination_reason in Multipart ?

theJC · 2026-02-14T07:26:11Z

Awesome, this looks to be a nice improvement to TRACE observability into subscription/defer activity !!!

An additional importantly needed observability is to have instruments/metrics emanated for the important un-happy conditions,

ie:

count of subscriptions failed due to hitting the max_open_subscriptions limit
count of stream_end / subgraph closing unexpectedly (by subgraph name)
count of client_disconnect (by client name)
count of heartbeat_delivery_failed (by client name)

Its important to be able to track how often these occur to track overall health of the subscription activity happening with Router, so anomalies in these trends can be detected and then acted upon.

…aph_error

mabuyo

@pragl 's comments were resolved so approving on behalf of Docs!

…ened_subscriptions` limit has been reached

…t-payload-errors

...urce/routing/observability/router-telemetry-otel/enabling-telemetry/standard-instruments.mdx

docs/source/routing/operations/subscriptions/configuration.mdx

rohan-b99 · 2026-02-19T17:57:37Z

Awesome, this looks to be a nice improvement to TRACE observability into subscription/defer activity !!!

An additional importantly needed observability is to have instruments/metrics emanated for the important un-happy conditions,

ie:

count of subscriptions failed due to hitting the max_open_subscriptions limit

count of stream_end / subgraph closing unexpectedly (by subgraph name)

count of client_disconnect (by client name)

count of heartbeat_delivery_failed (by client name)

Its important to be able to track how often these occur to track overall health of the subscription activity happening with Router, so anomalies in these trends can be detected and then acted upon.

Thanks for your feedback on this! Appreciate the input around metrics - I've implemented these along with some other refactors to make termination reasons clearer

This comment has been minimized.

Sign in to view

rohan-b99 changed the title ~~Rohan b99/subscription observability heartbeat payload errors~~ Add apollo.subscription.end_reason attribute to subscription spans Feb 3, 2026

rohan-b99 added 4 commits February 4, 2026 15:21

Implement subscription end reason attribute

271e4fe

Add docs

fa9c653

Remove "complete" reason

bbcb1c1

Add end reasons for schema reloads and config reloads

b9ce87b

rohan-b99 force-pushed the rohan-b99/subscription-observability-heartbeat-payload-errors branch from a5c8dad to b9ce87b Compare February 4, 2026 15:21

rohan-b99 added 4 commits February 6, 2026 17:56

Add attribute for defer.end_reason

d3bb6a4

Merge branch 'dev' into rohan-b99/subscription-observability-heartbea…

22b1c4e

…t-payload-errors

Add changeset

21ca659

Merge branch 'rohan-b99/subscription-observability-heartbeat-payload-…

6069b79

…errors' of https://github.com/apollographql/router into rohan-b99/subscription-observability-heartbeat-payload-errors

carodewig reviewed Feb 10, 2026

View reviewed changes

apollo-router/src/protocols/multipart.rs Outdated Show resolved Hide resolved

Store end reason in Multipart struct, set attribute in Drop impl

cfd3954

carodewig reviewed Feb 11, 2026

View reviewed changes

rohan-b99 added 2 commits February 11, 2026 17:34

Make usage of detect_reload_end_reason clearer

b5da305

Merge branch 'dev' into rohan-b99/subscription-observability-heartbea…

de446c9

…t-payload-errors

rohan-b99 marked this pull request as ready for review February 12, 2026 14:34

rohan-b99 requested review from a team as code owners February 12, 2026 14:34

pragl reviewed Feb 12, 2026

View reviewed changes

rohan-b99 and others added 2 commits February 12, 2026 17:24

Apply suggestions from code review

6aadae3

Co-authored-by: Parker <parker.ragland@apollographql.com>

Change gracefully -> successfully in docs + comments

a0c3010

theJC reviewed Feb 14, 2026

View reviewed changes

Add counters for subscription end reasons, rename stream_end to subgr…

4975ff9

…aph_error

mabuyo approved these changes Feb 18, 2026

View reviewed changes

Add metric for when a subscription request is rejected as the `max_op…

9a2fae0

…ened_subscriptions` limit has been reached

rohan-b99 added 2 commits February 19, 2026 12:18

Merge branch 'dev' into rohan-b99/subscription-observability-heartbea…

594d24d

…t-payload-errors

Format

cf8e42e

rohan-b99 commented Feb 19, 2026

View reviewed changes

...urce/routing/observability/router-telemetry-otel/enabling-telemetry/standard-instruments.mdx Outdated Show resolved Hide resolved

rohan-b99 commented Feb 19, 2026

View reviewed changes

docs/source/routing/operations/subscriptions/configuration.mdx Outdated Show resolved Hide resolved

Apply style guide suggestions

2deda87

rohan-b99 changed the title ~~Add apollo.subscription.end_reason attribute to subscription spans~~ Add subscription/defer observability attributes and metrics Feb 19, 2026

Remove stream_end naming

b7d34ff

Conversation

rohan-b99 commented Feb 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Footnotes

Uh oh!

apollo-librarian bot commented Feb 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Docs preview ready

Uh oh!

This comment has been minimized.

carodewig left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

carodewig left a comment

Choose a reason for hiding this comment

Uh oh!

carodewig Feb 11, 2026

Choose a reason for hiding this comment

Uh oh!

rohan-b99 Feb 11, 2026

Choose a reason for hiding this comment

Uh oh!

rohan-b99 commented Feb 12, 2026

Uh oh!

pragl left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

theJC Feb 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

theJC Feb 14, 2026

Choose a reason for hiding this comment

Uh oh!

theJC commented Feb 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mabuyo left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

rohan-b99 commented Feb 19, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Comments

rohan-b99 commented Feb 3, 2026 •

edited

Loading

apollo-librarian bot commented Feb 3, 2026 •

edited

Loading

theJC Feb 14, 2026 •

edited

Loading

theJC commented Feb 14, 2026 •

edited

Loading